To solve the gesture segmentation deviation problem under the interference of other skins and overlapping objects, a method of using depth data and skeleton tracking to segment gesture accurately was proposed. The minimum circumscribed circle, the average and the maximal inscribed circle of convexity defect, were combined to improve the detection of palm and the palm region's radius of various gesture. A fingertip candidate set was got through integrating the finger arc with convex hull, then real fingertips were obtained with three-step filtering. Six gestures have been tested in four transform cases, the recognition rate of flip, parallel, overlapping are all higher than 90% but the rate decreases obviously when tilting more than 70 degree and yawing more than 60 degree. The experimental results show that the accuracy of the proposed method is high in a variety of real scenes.